Worksheet 03: Critical Analysis - Level up

Methods and results

Get the methods section of your Target Paper on screen

I imagine you’ve read quite a few methods sections in your time. It may only be recently where your perspective has changed to actually reproducing the study’s methods. I believe this is a dramatic change in perspective. And one that forces a re-appraisal of quality. Now, without assuming too much, how successfully could you replicate your chosen paper’s study? It’s ok if you don’t fully understand any highly technical aspects, such as fMRI procedures or any descriptions of delivery or stimulus presentation, such as computer programmes or survey software (that’s for week 6!) Think more carefully about each of the following aspects: 1. Design of the study 2. Participants and recruitment 3. Materials 4. Procedure

Design

How easily can you identify the design? It may not be a fully experimental design. It may be quasi-experimental or even correlational. What is the design? What are the IVs? Are these between- or within-subject? Or a mix? What is/are the DV(s)?

Participants

Who took part in the study? Numbers, background, age range and demographics? Where did they come from? How heterogeneous a sample is it? Do you see any problems with the sample that might impact the results? Does this sound like a ‘high-quality’ sample? Does this sample appear to generalize to the entire population of interest? Were there any exclusion criteria? Or inclusion criteria? Was there any deception or information about the study retained in recruitment? This may only come to light in the procedure section. Was this necessary? If you had to replicate this exact sample, how easy would that be to do? If you could make the sample ‘better’ in any way, what would you do to achieve this? Does the paper mention how participants were allocated to condition or ‘categorised’. Did the researchers perform a Power Calculation to estimate a suitable sample size? We will be talking about Power Calculations next week.

Materials

You will need ‘materials’ to run your study. Many of you will build upon the tools used in the papers you critique. Now is an excellent time to start looking at these in detail. Does the research use any questionnaires or psychometric measures? Can you identify these? Can you FIND them? It’s always important to carefully consider the heritage and development of any questionnaires you use, either as a means of categorizing participants into groups (e.g. based on Anxiety). In future lectures we go in to more detail on Psychometrics and Scale design, so thinking about this is useful. Are there any tasks employed? How are they designed and delivered? Is any stimulus material described? If music or faces are used, where are these sourced, and how are they selected? Would you be able to replicate all the necessary stimulus materials? Can you be confident that you have similar stimulus sets after this replication process? Are you able to, for example, correctly replicate a stimulus set of Happy and Sad music? Or Angry and Fearful faces? Can you find enough to build the entire experiment? Are any of the materials piloted to make sure they work? Are there any manipulation or attention checks? What about awareness of research hypotheses? Could a participant identify these? Are any of the materials included with the paper? Do they have Supplementary Materials or Open Materials*? Is there any specialist equipment involved? Is there anything particular about the physical space employed if testing f2f? This may be relevant for next year. It is a good idea to think about the challenges of running a study online. Can you capture the appropriate data? If your chosen study includes anything like an intervention, or presentation of music or visual stimulus, how confident are you this is done well? How was the intervention or experimental condition developed?

Procedure

Can you determine when and how all the materials are put to use? Was there room for the experimenter to exert any bias? Was there consistency between experimenters or confederates? How was this achieved? What is the order of the various parts of the study? Is this the same for all participants? Are any elements counterbalanced? If so, how? If you had to estimate how long the procedure lasts, how long would it be? Do you get a sense that participants were looked after properly? Do you get a sense that optimal performance was obtained from the participants? Thinking about the duration and complexity of the task, to what extent is fatigue an issue? Is there a chance that performance may change in other ways over time? If a control condition was used, how closely did this match the experimental condition? If data are collected in a way dependent on the participant (e.g diary study, “over the course of a week” between sessions), how reliable is the resultant data?

Closing considerations

You have taken part in lots of experiments. The chances are, you were an excellent participant. How do you think the majority of participants may respond? How closely do you think they follow instructions? Running studies online can be tricky, but can offer new opportunities. Can you think of any obstacles? What about potential benefits of online research? You won’t be able to pay participants to take part in your research. How are you going to get the best performance out of them? How would you like to be treated as a participant in your first experiment?

Reuse